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Aided  Targeting  System  Simulation  Evaluation 


JOE  DE  MAIO*  AND  CURTIS  BECKERt 
Ames  Research  Center 


Summary 

Simulation  research  was  conducted  at  the  Crew  Station 
Research  and  Development  Facility  on  the  effectiveness 
and  ease  of  use  of  three  targeting  systems.  A  manual 
system  required  the  aviator  to  scan  a  target  array  area 
with  a  simulated  second  generation  forward  looking 
infrared  (FLIR)  sensor,  locate  and  categorize  targets  and 
construct  a  target  hand-off  list.  The  interface  between  the 
aviator  and  the  system  was  like  that  of  an  advanced  scout 
helicopter  (manual  mode).  Two  aided  systems  detected 
and  categorized  targets  automatically.  One  system  used 
only  the  FLIR  sensor,  and  the  second  used  FLIR  fused 
with  Longbow  radar.  The  interface  for  both  was  like  that 
of  an  advanced  scout  helicopter  aided  mode.  Exposure 
time  while  performing  the  task  was  reduced  substantially 
with  the  aided  systems,  with  no  loss  of  target  hand-off  list 
accuracy.  The  fused  sensor  system  showed  lower  time  to 
construct  the  target  hand-off  list  and  a  slightly  lower  false 
alarm  rate  then  the  other  systems.  A  number  of  issues 
regarding  system  sensitivity  and  criterion,  and  operator 
interface  design  are  discussed. 

1.  Introduction 

The  purpose  of  the  simulation  was  to  demonstrate  and 
evaluate  the  potential  performance  benefits  that  might  be 
achieved  through  the  use  of  an  aided  target  processing 
capability.  The  immediate  issue  was  the  relative  effect 
of  system  sensitivity  and  detection  criterion  on  total 
accuracy,  execution  timeline,  and  operator  acceptance. 
Two  generic  systems  were  investigated,  a  fused  sensor 
system  combining  forward-looking  infrared  (FLIR)  and 
radar  sensors  and  a  single  sensor,  FLIR  system. 

We  focused  on  two  design  parameters  of  aided  targeting 
systems,  sensitivity  (d')  and  criterion  (b).  Sensitivity  is  of 
interest  because  it  is  a  parameter  which  is  substantially 
improved  when  two  sensors  operating  in  different  spectral 
bands  are  fused  by  the  aided  targeting  system.  Criterion  is 
of  interest  because  the  relative  frequency  of  false  alarms 
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can  greatly  affect  the  usability  and  user  acceptance  of  a 
detection  system. 

The  sensitivity  of  a  fused  sensor  system  exceeds  that  of  a 
single  sensor  system  both  because  the  radar  provides 
more  accurate  range  data  and  because  the  fused  sensor 
system  uses  multiple  data  sources.  A  single  sensor  system 
operates  using  only  coarse,  target  range  data  obtained 
either  by  lasing  to  the  center  of  a  FLIR  frame  or  by 
triangulation.  These  relatively  poor  range  data  can  lead  to 
template  matching  ambiguities.  Radar  range  data  for 
individual  target  objects  can  eliminate  many  potentially 
erroneous  template  matches.  Further  ambiguity  reduction 
comes  because  voting  schemes  can  take  advantage  of 
correlations  in  the  data  from  the  two  sensors. 

An  important  reason  for  increasing  a  system’s  sensitivity 
is  to  reduce  the  rate  or  absolute  number  of  false  alarms  it 
produces.  False  alarms  by  the  aided  system  increase 
operator  workload,  and  they  also  lower  performance  by 
impairing  the  operator’s  vigilance.  For  any  system  the 
probability  of  detection  must  be  traded  off  against  the 
false  alarm  rate  to  provide  the  greatest  possible  number  of 
detections  without  impairing  operator  performance. 

Another  factor  of  interest  in  the  test  was  the  density  of 
target  objects  in  the  environment.  More  target  objects 
mean  more  detections.  More  detections  increase  operator 
tasking,  and  they  may  alter  the  way  in  which  the  system 
presents  detections  to  the  operator.  Our  concern  was  to 
determine  simply  whether  target  object  density  is  a 
concern  in  evaluating  aided  targeting  systems. 

While  we  understand  that  systems  such  as  those  evaluated 
here  are  intended  for  use  on  a  wide  variety  of  tasks  per¬ 
formed  under  a  broad  range  of  conditions,  we  did  not 
perform  a  parametric  evaluation  of  aided  targeting 
systems  over  their  full  range  of  application.  Rather  we 
selected  a  likely  combination  of  task  and  environment 
which  would  be  amenable  to  performing  an  evaluation  in 
simulation.  The  characteristics  of  the  test  environment 
included  high  thermal  contrast  (high  A  T),  giving  a  high 
quality  FLIR  image,  and  stationary  targets.  These 
conditions  permitted  good  performance  by  both  the 
aviator  and  the  system  and  an  adequate  level  of 
experimental  control. 
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2.  Method 

Experimental  Design 

The  simulation  was  designed  to  highlight  the  effects  of 
automation  on  target  detection  and  categorization  per¬ 
formance.  Two  aided  systems  were  studied.  One  was  a 
multi-spectral  system  using  passive  forward-looking 
infrared  (FLIR)  and  active  radar.  The  other  was  a  single 
sensor  system  using  only  FLIR.  The  actual  systems  would 
differ  in  a  number  of  performance  characteristics.  These 
characteristics  include  detection  sensitivity,  ability  to 
categorize  targets,  ability  to  provide  accurate  range  data, 
susceptibility  to  environmental  effects,  and  signature 
magnitude.  The  present  study  focused  only  on  detection 
sensitivity  and  target  categorization. 

Targeting  System  Modeling 

The  end-to-end  performance  of  a  detector  is  determined 
by  its  sensitivity  and  by  its  signal  detection  criterion,  and 
both  factors  must  be  considered  in  evaluating  systems. 

The  P(d)  is  the  probability  that  a  given  target  is  detected 
on  a  given  scan.  For  a  specified  system  sensitivity,  P(d)  is 
determined  only  by  the  detection  criterion.  A  high  cri¬ 
terion  leads  to  a  low  P(d).  In  effect  desired  P(d)  defines 
the  criterion  for  each  system.  When  the  criterion  has  been 
set  to  achieve  a  desired  P(d),  sensitivity  determines  FAR. 
A  lower  sensitivity  system  produces  more  false  alarms  for 
a  given  P(d)  than  does  a  higher  sensitivity  system.  A 
notional  illustration  of  the  relationship  between  the 
sensitivity  of  the  detector  (d'),  P(d)  and  FAR  is  presented 
in  the  receiver  operating  characteristic  (ROC)  shown  in 
figure  2-1 .  The  ROC  relates  the  P(d)  of  a  detector  to  its 


FAR.  Since  these  two  parameters  vary  with  criterion,  a 
sufficiently  high  criterion  will  result  in  no  detections  and 
no  false  alarms  and  a  sufficiently  low  criterion  will  result 
in  a  P(d)  of  1 .0  and  a  probability  of  false  alarm  given  no 
target  of  1.0.  Between  these  extremes,  detections  and 
false  alarms  will  vary  to  produce  the  monotonic  function 
shown  in  figure  2-1.  The  degree  of  curvature  of  the  ROC 
is  proportional  to  d'. 

In  the  present  study,  two  levels  of  correct  detection 
performance  were  set.  The  probability  of  detection  (P(d)) 
determined  criterion  based  on  system  sensitivity,  and  a 
corresponding  false  alarm  rate  was  determined.  Thus 
system  sensitivity  and  signal  detection  criterion  were 
covaried  to  yield  four  representative  combinations  of 
correct  detections  and  false  alarms,  specified  on  a  per 
scan  basis.  With  a  scan  field  of  regard  (FOR)  of  6°  x  45° 
centered  2°  below  the  horizon,  the  correct  detection  and 
false  alarm  rates  (FAR)  shown  in  table  2-1  obtain. 

“Target  objects”  were  placed  in  the  data  base  to  be 
detected,  with  the  number  of  target  objects  varied  across 
vignettes  (see  table  2-4  in  Operator  Interface  section). 
“Non-target  objects”  were  placed  in  the  data  base  to  yield 
the  false  alarm  rates  per  scan  shown  in  table  2-1.  FAR  is  a 
function  of  system  sensitivity  and  detection  criterion. 

The  P(d)  shown  in  table  2-1  is  an  average  over  all  targets 
(target  and  non-target  objects  are  treated  identically  by 
the  targeting  system).  The  P(d)  for  an  individual  object 
was  determined  by  a  function  of  P(d)j^j  (see  eq.  2-1),  the 
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Figure  2- 1.  Sample  receiver  operating  characteristic. 


Table  2-1.  Targeting  system  configurations 


High  sensitivity 

Low  sensitivity 

High 

P(d)  =  0.6 

P(d)  =  0.6 

criterion 

FAR  =  0.6 

FAR  =  3.6 

Low 

P(d)  =  0.9 

*0 

/- — X 

CL 

It 

o 

vO 

criterion 

FAR  =  4.5 

FAR  =  19.8 
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average  P(d),  range  and  a  detectability  factor  (three 
levels).  Tanks  had  a  detectability  factor  of  0.02.  APCs 
and  tracked  ADUs  had  a  detectability  factor  of  0.0. 
Wheeled  vehicles  had  a  detectability  factor  of  -0.02. 
Similarly  non-target  objects  had  a  detectability  factor, 
0.02  for  living  (cow),  0.0  for  man-made  (dead  vehicles), 
-0.02  for  non-living  (rocks  and  dead  trees).  The  range 
scale  factor  was  0.08  at  minimum  range  (about  1  km)  and 
0.0  at  2  km.  Individual  probability  of  detection  was  the 
sum  of  the  average  P(d),  detectability,  and  range  scalar. 
The  range  of  individual  P(d)  was  0.5  to  0.7  for  the  high 
criterion  conditions  and  0.8  to  1.0  for  the  low  criterion 
conditions.  Detectability  factors  associated  with  each 
type  of  target  object  and  non-target  object  are  shown  in 
table  2-2.  Objects  were  placed  in  the  data  base  to  give  an 
average  detectability  factor  across  all  objects  of  0.  This 
scheme  allows  variation  in  P(d)  independent  of  average 
P(d).  Eighty  percent  of  the  variation  in  P(d)  within  a 


scan  results  from  range  effects,  while  20%  results  from 
variation  in  the  targets.  The  P(d)  of  object,  was  given  by 
the  following  formula: 

P(d)i=P(d)M  +  d  +  r  (2-1) 

where 

d  =  {-0.02, 0.0, +0.02} 

r  =  -0.08  +  {0.16  *  [(range,  -  ranges,,)  /  (range  max 

-range  min)]) 

The  sensor  model  was  intended  to  provide  a  qualitatively 
correct  rendering  of  the  effects  of  target  type  and  range. 
The  target  detection  systems  were  modeled  through  a 
simple  Monte  Carlo  process.  Each  target  in  the  system 
FOR  was  detected  through  an  independent  stochastic 
process  using  P(d)  determined  by  equation  2-1.  Target 
objects  and  non-target  objects  were  detected  according  to 


Table  2-2.  Experimental  targets 


Target  type 

Classification  type 

Recognition  type 

Detectability  factor 

Ml 

Tracked 

Tank 

0.02 

M60 

Tracked 

Tank 

0.02 

M3 

Tracked 

APC 

0.0 

M113 

Tracked 

APC 

0.0 

HUMMV 

Wheeled 

Truck 

-0.02 

T72 

Tracked 

Tank 

0.02 

BTR 

Wheeled 

Truck 

-0.02 

BMP 

Tracked 

APC 

0.0 

ZSU 

Tracked 

ADU 

0.0 

SA9 

Wheeled 

ADU 

-0.02 

Zil 

Wheeled 

Truck 

-0.02 

Rock 

- 

- 

-0.02 

Rock 

- 

- 

-0.02 

Rock 

- 

- 

-0.02 

Dead  truck 

- 

- 

0.0 

Dead  tree 

- 

- 

-0.02 

Dead  hind 

- 

- 

0.0 

Dead  tank 

- 

- 

0.0 

Cow 

- 

- 

0.02 
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the  same  stochastic  process.  Appropriate  FARs  were 
obtained  by  varying  the  number  of  non-target  objects 
placed  in  the  data  base.  In  the  experimental  context  the 
differences  in  individual  P(d)  provided  a  basis  for  target 
prioritization  as  well  as  the  detection  performance  of  the 
systems.  If  a  target  were  detected,  the  individual  P(d) 
would  be  used  for  target  prioritization,  with  higher  P(d) 
targets  presented  first. 

The  aided  targeting  systems  provided  information  about 
target  type.  The  single  sensor  system  indicated  target 
classification.  The  fused  sensor  system  provided  recog¬ 
nition  information.  The  target  labels  used  are  shown  in 
table  2-3.  The  probability  of  correct  categorization 
conditional  on  target  detection  was  0.6  for  both  systems. 


Table  2-3.  Aided  system  categorization  output 


Single  sensor 
classification 

Multi-sensor 

recognition 

Tracked 

Tank 

Wheeled 

Truck 

Unknown 

ADU 

APC 

Unknown 

A  manual  targeting  system  was  included  as  a  baseline  for 
comparison  against  the  aided  systems.  The  manual  system 
used  the  same  FLIR  and  laser  ranging  system  as  the 
single  sensor  aided  system,  but  these  subsystems  were 
under  manual  control.  The  FLIR  sensor  had  a  selectable 
field  of  view  (FOV)  of  1 .5°  x  2°  (narrow)  or  6°  x  8° 
(medium).  The  laser  range  finder  was  triggered  manually 
to  obtain  a  range  for  each  target  included  in  the  target 
hand-off  list.  Since  the  manual  system  detected  no  targets 
and  generated  no  false  alarms,  there  could  be  no  specified 
FAR,  but  we  assumed  that  in  the  worst  condition  the 
aided  system  detected  every  possible  non-target  object  in 
the  environment.  Therefore,  the  non-target  object  count 
for  single  sensor,  low  criterion  condition  was  used  for  the 
manual  condition. 

Target  environment-  Real  targets  were  computer 
generated  models  of  the  military  vehicles  listed  in 
table  2-2.  Objects  were  placed  into  correlated  out-the- 
window  (OTW)  visual  and  FLIR  data  bases.  The  OTW 
scene  was  created  on  a  General  Electric  Compu-scene  IV 
image  generator.  The  FLIR  imagery  was  generated  on 
a  Silicon  Graphics  Reality  Engine.  Vehicles  were  sta¬ 
tionary,  and  their  placement  conformed  to  tactics  and 


doctrine  as  reflected  in  the  Rotorcraft  Pilot’s  Associate 
vignettes.  There  were  four  conditions  of  target  environ¬ 
ment,  differing  in  the  number  of  real  targets  placed  in  the 
environment.  A  target-free  condition  contained  no  target 
objects,  and  the  only  detections  reported  by  the  aided 
systems  were  non-target  objects.  A  low  target  density 
condition  corresponded  to  a  combat  reconnaissance 
platoon.  This  condition  contained  six  targets.  A  medium 
density  condition,  corresponding  to  a  regimental  recon¬ 
naissance  company,  contained  10  targets.  The  high  target 
density  condition  contained  14  targets.  This  condition 
corresponded  to  an  advanced  guard. 

The  experimental  design  is  shown  in  table  2-4.  Two 
factors  in  the  table  taken  together  define  the  aided 
targeting  system.  The  P(d)  identically  determines  the  hit 
rate,  and  it  plus  the  number  of  non-target  objects  define 
the  false  alarm  rate  (number  of  non-target  objects 
detected  on  a  scan).  A  second  environmental  factor, 
number  of  target  objects,  defined  the  operational  environ¬ 
ment  of  the  test.  Four  target  object  densities  were 
presented. 


Operator  Interface 

Targeting  display  page-  The  targeting  task  was 
performed  on  a  display  page  presented  on  the  tactical 
situation  display  (TSD)  of  the  CSRDF.  The  TSD  consists 
of  a  12  inch  high-resolution,  color  CRT  with  an  infrared 
touch  screen.  Subjects  touched  the  area  of  the  display 
showing  the  desired  softkey  control  to  activate  the 
control.  Touchscreen  controls  were  of  three  types, 
instantaneous,  on/off,  and  alternate  action.  In  the 
following  discussion  controls  are  indicated  by  BOLD 
CAPITAL  letters. 

The  targeting  display  page  consisted  of  a  system  control 
area  along  with  imagery  and  controls  specific  to  the 
targeting  system.  The  system  control  area  for  targeting 
systems  is  shown  in  figure  2-2.  This  area  would  allow  for 
selection  of  manual  sensor  scan  and  aided  scan  if  this 
function  were  supported.  Top  level  controls  were 
continuously  displayed  around  the  side  of  the  targeting 
display  page.  Imagery  and  lower  level  controls  were 
displayed  in  the  center.  Labeling  controls  were  displayed 
at  the  bottom.  For  the  purposes  of  the  present  test,  manual 
and  aided  scan  modes  were  mutually  exclusive.  Therefore 
the  controls  for  selecting  one  or  the  other  were  active  only 
if  that  test  condition  were  available.  Since  switching 
between  modes  was  not  an  issue,  highlighting  of  available 
modes  was  not  provided. 

Manual  system  controls -The  MANUAL  FLIR  softkey 
activated  the  sensor  slewing  control  for  manual  mode 
targeting.  Two  sensor  fields  of  view  (FOV)  were  selec- 


4 


table  by  a  toggling  softkey,  MEDIUM  (6°  x  8°)  and 
NARROW  (1.5°  x  2°).  The  right  side  hand  controller 
allowed  the  aviator  to  slew  the  sensor  line  of  sight  left 
and  right  and  up  and  down  over  a  range  of +1 10°  in 
azimuth  and  +207-60°  in  elevation.  An  arrow  in  the 
lower  left  corner  indicated  the  sensor  line  of  sight  in 
azimuth  relative  to  the  armament  datum  line  (ADL).  The 
aviator  selected  a  target  by  touching  the  image  on  the 
sensor  image  display,  and  the  system  responded  by 


placing  a  complete  box  around  the  target  with  the  label 
“UNK”  (unknown,  fig.  2-3)  and  bringing  up  first  level 
(classification)  LABELING  controls  (fig.  2-4).  The 
aviator  could  label  the  target  at  this  level  or  select 
REC(ognition)  or  ID(entification).  The  aviator  selected 
a  label  from  the  set  shown  in  figure  2-4.  Previously 
accepted  targets  that  were  not  currently  selected  were 
indicated  by  a  broken  box. 


Table  2-4.  Experimental  conditions 


Sensor/ 

criterion13 

Target 

density 

P(d) 

Average 

targets 

detected 

(hits) 

Average 

number 

false 

alarms 

Number 

real 

targets 

Number 

false 

targets 

Number 

total 

targets 

F/H 

None 

0.6 

0 

0.6 

0 

1 

1 

F/H 

Low 

0.6 

3.6 

0.6 

6 

1 

7 

F/H 

Medium 

0.6 

6 

0.6 

10 

1 

11 

F/H 

High 

0.6 

8.4 

0.6 

14 

1 

15 

F/L 

None 

0.9 

0 

4.5 

0 

5 

5 

F/L 

Low 

0.9 

5.4 

4.5 

6 

5 

11 

F/L 

Medium 

0.9 

9 

4.5 

10 

5 

15 

F/L 

High 

0.9 

12.6 

4.5 

14 

5 

19 

S/H 

None 

0.6 

0 

3.6 

0 

6 

6 

S/H 

Low 

0.6 

3.6 

3.6 

6 

6 

12 

S/H 

Medium 

0.6 

6 

3.6 

10 

6 

16 

S/H 

High 

0.6 

8.4 

3.6 

14 

6 

20 

S/L 

None 

0.9 

0 

19.8 

0 

22 

22 

S/L 

Low 

0.9 

5.4 

19.8 

6 

22 

28 

S/L 

Medium 

0.9 

9 

19.8 

10 

22 

32 

S/L 

High 

0.9 

12.6 

19.8 

14 

22 

36 

MAN 

None 

n/a 

n/a 

n/a 

0 

22 

22 

MAN 

Low 

n/a 

n/a 

n/a 

6 

22 

28 

MAN 

Medium 

n/a 

n/a 

n/a 

10 

22 

32 

MAN 

High 

n/a 

n/a 

n/a 

14 

22 

36 

a“F”  and  “S”  refer  to  fused  sensor  and  single  sensor  aided  systems.  “MAN”  refers  to  the  manual  targeting 


system.  “H”  and  “L”  refer  to  high  and  low  detection  criteria. 
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Figure  2-2.  System  control  area  of  targeting  display  page. 
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Figure  2-3.  Manual  system  targeting  display  page. 


Aided  system  control  and  display-  The  base  control 
page  contained  softkeys  for  operating  the  aided  targeting 
systems.  INIT  SCAN  caused  the  system  to  execute  a  two 
pass  scan  (near  and  far  bar).  For  the  present  test  the  area 
scanned  was  always  22.5°  left  and  right  of  the  ADL  and 
1°  above/50  below  it,  so  no  scan  limit  controls  were 
provided. 

Following  scan  completion,  the  REVIEW  softkey 
allowed  the  aviator  to  review  the  system  detections.  Four 
review  mode  controls  were  available  on  the  right  side  of 
the  page.  CHIPS  allowed  the  aviator  to  view  up  to 
18  detections,  presented  in  groups  of  six  (fig.  2-5).  The 
P(d)  of  each  was  used  to  prioritize  detections,  with 
highest  priority  (highest  P(d))  shown  in  chip  number  one. 
Detections  with  priority  of  19  or  below  were  not  pre¬ 
sented  in  chips  mode.  Three  frames  modes  allowed  the 
aviator  to  view  each  of  the  92  FLIR  frames  produced  by 
the  scans,  with  detections  highlighted  (fig.  2-6).  AUTO 
FRAME  first  presented  the  frame  in  the  upper  left.  At 
20  second  intervals  the  system  automatically  stepped  one 
frame,  moving  left  to  right  and  down.  Any  input  by  the 
aviator  reset  the  interval  counter.  TGT  FRAME  per¬ 
formed  similar  stepping  but  only  between  frames  contain¬ 
ing  detections,  beginning  with  the  upper,  left-most  frame 
containing  a  detection.  MAN  FRAME  did  not  step 
automatically,  but  the  aviator  could  step  manually  using 
the  UP,  DOWN,  LEFT,  and  RIGHT  softkeys.  These 
keys  were  also  active  in  the  automatic  stepping  modes. 

A  grid  displayed  below  the  sensor  imagery  showed  the 
frames  array.  The  currently  displayed  frame  was  filled. 
Frames  containing  detections  were  stippled. 

Target  selection  and  labeling-  A  target  chip  could  be 
selected  by  touching  the  chip,  after  which  YES  or  NO 
would  accept  or  reject  the  detection.  Alternatively  YES 
or  NO  could  be  touched  to  select  and  accept  or  reject 
simultaneously.  Label  controls  and  a  system-determined 
categorization  became  available  when  a  detection  was 
accepted.  The  initial  level  of  categorization  was  classifi¬ 
cation  for  the  single  sensor  system  and  recognition  for  the 
fused  sensor  system. 

In  frames  mode  the  bottom  comers  of  a  box  indicated  a 
system  detection,  a  broken  box  indicated  a  detection  that 
had  been  accepted,  and  a  complete  box  indicated  the 
selected  target.  A  bullet  (•)  indicated  an  object  that  the 
aviator  had  rejected  as  a  target.  A  target  object  could  be 
selected  by  touching  it  whether  the  system  had  detected  it 
or  not.  If  the  system  had  detected  it  or  it  had  previously 
been  accepted,  it  could  also  be  selected  using  the  PREV 
TGT  and  NEXT  TGT  softkeys.  Label  controls  were  the 
same  as  for  chips  mode. 

Target  hand-off  list-  Objects  labeled  by  the  aviator  as 
target  objects  or  non-target  objects  were  automatically 


placed  in  a  target  hand-off  list.  This  list  was  displayed  on 
a  6  inch  CRT  located  on  the  right  side  of  the  cockpit.  For 
each  object  in  the  list  the  display  presented  a  sequence 
number,  the  object  label,  and  the  object  location  (in 
Universal  Transverse  Mercator  coordinates).  Up  to  five 
target  entries  could  be  displayed  at  one  time.  PAGE  UP 
and  PAGE  DN  moved  the  display  up  and  down  the  list  in 
groups  of  five.  “A”  and  “V”  moved  up  and  down  the  list 
one  object  at  a  time.  The  SEND  softkey  terminated  the 
experimental  trial. 

Experimental  Task 

The  task  presented  to  the  experimental  subjects  was  to 
reconnoiter  a  target  area  for  which  they  have  no  prior 
intelligence  and  to  formulate  and  transmit  a  target  hand- 
off  list.  An  experimental  trial  began  at  a  point  along  the 
ingress  route  to  the  target  area.  The  terminal  phase  on  the 
ingress  (about  one  minute)  was  prerecorded,  and  the 
experimental  aircraft  flew  the  subject  to  the  designated 
observation  point.  The  subject  wore  the  fiber-optic  helmet 
mounted  display  (FOHMD)  to  provide  out-the-window 
cues  for  orientation  to  the  task  environment.  At  the  end  of 
the  ingress,  he  called  for  an  unmask,  and  the  aircraft 
executed  a  prerecorded  unmask  to  a  hover.  The  aircraft 
was  oriented  with  the  target  area  directly  in  front  of  it. 

The  subject  executed  his  targeting  task.  In  the  aided 
conditions  this  activity  consisted  of  initiating  the  scan. 

The  subject  then  commanded  a  reipask  when  the  scan  was 
completed,  after  which  he  constructed  and  transmitted  the 
target  list.  The  process  of  constructing  the  list  consisted  of 
reviewing  the  output  of  the  aided  system,  accepting  or 
rejecting  each  detection  and  providing  categorization  to 
the  highest  level  possible.  The  trial  ended  at  message 
transmission.  In  the  manual  condition  the  subject  slewed 
the  FLIR  sensor  and  entered  target  data  and  transmitted 
the  target  hand-off  list.  The  trial  ended  with  message 
transmission. 

Subjects 

Fourteen  aviators  participated  in  the  experiment.  Two  test 
pilots  were  supplied  by  the  Night  Vision  and  Electronic 
Sensors  Directorate  (NVESD).  These  two  aviators  served 
from  07  SEP  to  1 1  SEP  94.  Their  primary  role  was  to 
support  final  debug  of  simulation  and  procedures,  and 
their  data  were  not  analyzed.  Twelve  subjects  were 
operational  AH-64  aviators.  These  aviators  had  no 
CSRDF  experience,  but  they  were  experienced  in  the 
AH-64  combat  mission  simulator.  These  aviators  served 
in  three  groups  of  four  each.  Each  group  spent  one  week 
at  the  CSRDF  during  the  period  from  1 2  SEP  through 
02  OCT  94.  Experience  of  the  12  aviators  is  shown  in 
table  2-5. 


8 


Target  object  labeling  controls. 


Figure  2-5.  Chips  mode  display. 
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Figure  2-6.  Frames  mode  display. 


Tabic  2-5.  Experimental  aviator  operational  experience 


Aviator 

ID 

AH-64 

front  seat  time 
(hr) 

AH-64 
back  seat 
time  (hr) 

Attack 
helicopter 
time  (hr) 

Total 
helicopter 
time  (hr) 

1 

200 

150 

600 

750 

2 

160 

40 

500 

580 

3 

300 

70 

450 

700 

4 

300 

300 

2000 

3500 

5 

250 

400 

1500 

1700 

6 

200 

50 

300 

350 

7 

150 

20 

170 

350 

8 

150 

30 

300 

465 

9 

150 

50 

475 

550 

10 

50 

50 

1600 

2000 

11 

85 

40 

500 

700 

12 

170 

20 

190 

350 

Procedure 

Data  collection  took  place  over  three  days  for  each  group 
of  aviators.  Each  aviator  performed  two  data  collections 
per  day  (roughly  two  hours  total),  being  exposed  once  to 
each  system  X  target  density  condition.  Prior  to  starting 
the  data  collection,  the  aviators  received  training  in  the 
operation  of  the  simulated  targeting  systems,  and  they 
were  fitted  with  a  custom  liner  for  the  FOHMD.  Aviators 
were  trained  in  the  operation  of  the  targeting  systems  as 
well  as  in  recognition  of  the  CSRDF  FLIR  imagery  of 
target  ground  vehicles.  Aviators  were  trained  until  they 
and  the  test  administrator  agreed  that  they  were  proficient. 

Data  collection  took  place  on  Tuesday,  Wednesday  and 
Thursday  of  each  week.  Two  aviators  were  scheduled 
each  morning  session,  and  two  were  scheduled  each 
afternoon.  Each  session  consisted  of  two  one-hour  blocks 
per  subject.  Each  subject  completed  two  replications  of 
each  condition. 

Each  one-hour  block  consisted  of  10  data  collection  runs 
with  the  order  of  experimental  conditions  randomized.  On 
each  run  the  simulated  aircraft  was  initialized  a  short 
distance  from  the  observation  point  (OP).  The  targeting 
system  was  correctly  configured,  and  the  appropriate 
display  page  was  presented  on  the  back  seat  displays.  The 
right  hand  display  contained  the  target  list  display.  The 


center  display  contained  the  sensor  system  display.  The 
left  hand  display  contained  the  map.  The  aviator  also 
wore  the  FOHMD  to  aid  in  orienting  to  the  tactical 
environment,  but  neither  the  OTW  scene  nor  helmet 
displayed  symbology  was  used  in  the  targeting  task. 

When  the  aviator  signaled  that  he  was  ready,  the  pre¬ 
recorded  run-in  to  the  OP  was  executed.  Data  collection 
began  when  the  aircraft  achieved  a  masked  position  at  the 
OP.  The  aviator  signaled  readiness  to  begin  the  targeting 
task.  A  prerecorded  unmask  to  a  hover  was  executed. 

The  prerecorded  flight  sequence  was  frozen  for  as  long 
as  the  aviator  needed  to  complete  his  task,  and  then  the 
sequence  resumed  with  a  remask.  The  aviator  completed 
the  targeting  task  and  transmitted  the  target  list.  Data 
recording  terminated  with  transmission  of  the  target  list, 
and  the  aviator  provided  a  subjective  evaluation  of  the 
targeting  system  (see  appendix  A).  A  final  debriefing 
gave  aviators  an  opportunity  to  comment  on  all  aspects  of 
the  evaluation. 

Data  Collection  and  Analysis 

Real  time  objective  measures  of  performance  were 
collected  along  with  the  subjective  questionnaire  data. 
Data  on  correct  detections  (hits),  misses,  false  alarms, 
correct  rejections  and  categorization  accuracy  came  from 
the  target  hand-off  list.  An  explicit  action  to  reject  each 
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non-target  object  detection  (system  false  alarm)  was 
required  in  order  to  support  data  collection.  In  addition 
objective  measures  of  timing  were  determined.  These 
included  the  following: 


Time  to  accept  or  reject  each  target 
Total  time  to  target  hand-off 
Exposure  time 
Scanning  sequence 


3.  Results 

The  objective  of  the  simulation  was  to  examine  how 
system/configuration  changes  in  an  aided  targeting 
system  might  affect  the  utility  of  that  system.  In  order  to 
address  this  issue,  we  have  decomposed  the  issue  into 
three  component  questions.  These  questions  are  (1)  how 
does  aiding  affect  the  performance  of  the  aviator  on  the 
targeting  task,  (2)  how  do  changes  in  the  performance 
of  the  aiding  system  affect  aviator  performance,  and 
(3)  what  factors  affect  the  interaction  of  the  aviator  with 
the  system.  These  questions  were  addressed  by  way  of 
analyses  of  variance  on  several  dependent  measures.  A 
2x2  repeated  measures  ANOVA  looked  at  aided  system 
and  criterion  effects.  A  three-way  repeated  measures 
ANOVA  looked  at  aided  versus  manual  system.  A 
2x2x4  repeated  measures  ANOVA  looked  at  effects  of 
target  density  on  aided  system  performance.  ANOVA 
tables  are  presented  in  appendix  B.  Significant  effects  are 
shown  in  bold  for  objective  measures  (B1-B13).  Only 
significant  effects  are  shown  for  subjective  evaluations 
(B14-B18). 

The  presentation  of  results  is  organized  as  follows.  First 
we  discuss  the  effect  of  aiding.  Second  we  discuss  how 
sensitivity  of  the  aided  systems  affect  the  quality  of 
performance  and  speed.  Next  we  look  at  the  effects  of 
variation  in  criterion.  We  finish  with  a  discussion  of  the 
effects  of  target  density.  A  review  of  the  aviators’ 


subjective  evaluations  supplements  the  analysis  of  the 
objective  data. 


Effects  of  Aiding 

The  major  effect  that  could  be  attributed  to  aiding, 
per  se,  was  a  substantial  reduction  in  exposure  time 
(fig.  3-1).  For  this  analysis  exposure  time  was  defined 
to  be  the  period  of  time  during  which  there  was  an 
unobstructed  line  of  sight  from  the  ownship  to  a  vehicle 
in  the  target  array  being  scanned.  Exposure  time  with 
aiding  was  determined  by  the  time  required  to  enter  a 
stable  hover  above  the  masking  obstruction,  to  initiate 
the  sensor  scan  manually,  to  complete  the  automated 
scanning  process,  and  to  descend  below  the  obstruction. 
Exposure  time  in  the  manual  condition  was  determined  by 
the  transition  times  as  above  plus  the  time  required  to 
scan,  identify  targets,  and  build  the  target  hand-off  list. 
Mean  exposure  time  was  258  seconds  for  manual 
targeting.  Exposure  time  was  37  to  61  seconds  for  the 
aided  system,  a  reduction  of  over  75%.  There  was  no 
difference  between  aided  system  configurations. 

Detection  performance  was  measured  by  comparing  the 
target  hand-off  list  generated  by  the  aviator  with  ground 
truth  represented  in  the  simulation  data  base.  Detection 
performance  was  generally  very  good,  since  the  FLIR 
simulation  represented  optimal  viewing  conditions. 
Overall  the  rate  of  correct  detection  was  over  98%,  that  is, 
of  3095  target  objects  only  42  were  missed.  There  were 
no  differences  in  hit  or  miss  rates  for  the  various  systems 
(fig.  3-2).  The  fused  sensor  system  did  show  a  slightly 
lower  acceptance  rate  for  false  alarms  than  the  single 
sensor  and  manual  systems  (fig.  3-3).  Most  of  the  false 
alarms  were  non-target  vehicles  that  were  most  easily 
confused  with  target  objects  (fig.  3-4).  The  labeling  of  a 
non-target  object  as  a  target  by  the  aiding  system  did  not 
bias  the  aviator’s  response  to  the  non-target  objects  that 
were  not  visually  confusable. 
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Figure  3-1.  Exposure  time  for  each  targeting  system. 
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Figure  3-2.  Number  of  targets  not  included  in  target  hand-off  list.  For  manual  condition  2  x  N  number  is  comparable  to 
aided  systems  since  manual  sample  is  one-half  as  large. 
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TARGETING  SYSTEM 


Figure  3-3.  Number  of  false  alarms  included  in  target  hand-off  list. 
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Figure  3-4.  Number  of  non-target  objects  included  in  target  hand-off  list. 
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There  was  a  difference  in  the  reason  for  missed  targets  in 
the  aided  conditions  as  compared  to  the  manual  condition. 
In  a  number  of  cases,  target  objects  were  missed  because 
they  were  not  displayed  to  the  aviator  by  the  aided 
systems.  More  than  one-half  of  the  target  objects  missed 
with  the  aided  systems  were  never  presented  to  the 
aviator  (fig.  3-5).  Using  the  manual  system,  the  aviator 
was  presented  every  target  object  in  the  scene. 

Appendix  C  shows  a  typical  range  of  scan  fields  for 
the  aided  and  manual  systems. 

The  presentation  of  the  manual  scans  in  appendix  C  uses 
a  denser  pattern  of  dots  to  represent  the  narrow  field  of 
view  (FOV)  and  a  sparser  pattern  of  dots  to  represent  the 
medium  FOV.  When  scanning  manually,  the  aviators 
tended  to  concentrate  on  the  main  body  of  target  objects 
or  confusable  non-target  objects  using  the  narrow  FOV. 
They  searched  for  the  main  body  in  the  medium  FOV, 
and  they  also  used  this  FOV  to  search  for  outlying  target 
objects  after  finishing  the  main  body.  This  pattern  of 
use  is  reflected  in  the  small  area  of  dense  stippling  sur¬ 
rounded  by  sparse  stippling  shown  in  the  sample  scans. 
Between  narrow  and  medium  FOV  the  sample  scans 
cover  an  area  between  1 50  square  degrees  and  more  than 
400  square  degrees.  The  assigned  area  was  270  square 
degrees. 

The  presentation  of  the  aided  scans  indicates  target 
objects  by  a  “+,”  target  objects  displayed  in  chips  mode 
by  a  superimposed  circle  (“o”),  and  displayed  frames  by  a 
stippled  area.  Uncircled  target  objects  in  an  unfilled  area 
were  not  presented  to  the  aviator.  All  three  examples  of 
an  aided  scan  contain  undisplayed  target  objects. 

The  sequence  of  operation  with  the  aided  systems  was 
different  than  that  used  with  the  manual  system.  The 
aviators  began  by  verifying  all  the  detections  presented  in 


chips  mode.  If  there  were  more  than  1 8  detections,  they 
verified  the  remaining  detections  in  target  frame  mode. 
The  aviators  proceeded  to  subsequent  targets  by  stepping 
sequentially  left,  right,  up  and  down.  They  added  any 
target  objects  missed  by  the  system  that  they  encountered 
during  this  search  process.  After  completing  the  detec¬ 
tions,  they  made  a  search  for  outlying  undetected  target 
objects.  This  search  was  restricted  to  a  small  area  around 
the  main  body  of  target  objects,  owing  to  the  slowness 
and  heavy  manual  involvement  of  stepping  through 
frames.  The  sample  scans  cover  an  area  between 
48  square  degrees  and  slightly  over  100  square  degrees. 

Effects  of  Aided  Systems 

Differences  between  and  within  aided  systems  showed 
their  effects  in  the  time  needed  to  perform  targeting 
operations.  There  were  performance  time  differences 
resulting  from  differing  sensitivity  as  well  as  differences 
arising  from  the  type  of  operations  performed  by  the 
aviator. 

On  the  whole  the  fused  system  supported  much  faster 
completion  of  the  target  hand-off  list  than  did  either  the 
single  sensor  system  or  the  manual  system,  which  did  not 
differ  from  each  other.  Time  required  to  construct  the 
target  hand-off  list  (target  hand-off  time)  was  over 
50  seconds  less  with  the  fused  system  than  with  the  other 
systems.  This  represents  about  a  20%  reduction  (fig.  3-6). 
For  the  aided  systems,  target  hand-off  time  included  time 
to  process  the  chips  array  and  the  target  objects  presented 
in  frames  plus  the  time  to  search  for  outlying  target 
objects  plus  the  time  to  verify  the  target  hand-off  list.  The 
process  began  as  soon  as  the  system  presented  a  detected 
object  to  the  aviator.  Presentation  of  chips  began  during 
the  scan,  while  the  aircraft  was  still  unmasked,  but  most 
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Figure  3-5.  Proportion  of  missed  target  objects  not  presented  to  aviator. 
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of  the  time  was  spent  after  remask.  By  contrast  the  aviator 
performed  the  scanning  in  the  manual  condition,  so  the 
target  hand-off  time  is  equal  to  the  exposure  time. 

The  differences  in  target  hand-off  time  are  interesting 
because  they  reflect  not  simply  aiding  but  how  well  the 
aiding  is  done.  The  effectiveness  of  aiding  depends  on 
how  the  aviator  must  act  on  the  data  presented  by  the 
aiding  system.  The  target  hand-off  time  was  shorter  for 
the  fused  system  even  though  the  average  time  to  process 
an  individual  detection  was  shorter  for  the  single  sensor 
system  (fig.  3-7).  This  seeming  paradox  exists  because 
the  average  time  to  process  a  detection  masks  important 
individual  differences  between  detections.  The  time 
required  to  process  each  detection  is  determined  by  two 


factors,  object  type  and  presentation  mode.  Non-target 
objects  are  processed  faster  than  target  objects,  because 
they  require  no  labeling  of  target  type,  and  chips 
presentations  are  processed  faster  than  frames 
presentations  (fig.  3-8). 

When  the  (single  sensor)  aided  system  produces  more 
false  alarms,  the  aviator  must  reject  more  non-target 
objects.  Since  this  is  the  relatively  faster  process,  the 
average  time  per  object  decreases.  If,  however,  these 
objects  were  not  detected,  they  would  not  be  processed 
at  all.  Therefore  the  total  time  to  accomplish  the  task 
increases  by  the  amount  required  to  process  the  added 
detections. 
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Figure  3-6.  Time  to  construct  and  send  target  hand-off  list. 
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Figure  3-7.  Time  to  enter  individual  target  data. 
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Figure  3-8.  Time  to  enter  individual  target  data  by  system  mode  and  by  target  type. 


Increasing  the  number  of  false  alarms  also  increases  the 
total  number  of  detections.  When  there  are  more  than 
18  total  detections,  some  are  presented  only  in  frames 
mode.  Frames  mode  processing  is  slower  than  chips  mode 
processing.  Detected  target  objects  displaced  to  frames 
mode  presentation  take  longer  to  process,  so  the  total  time 
to  perform  the  task  increases. 

Chips  mode  processing  is  faster  than  frames  mode 
processing  because  the  former  allows  rapid  selection  of 
detections  for  processing  while  the  latter  requires  a 
cumbersome  search  sequence.  In  chips  mode  up  to  six 
detections  are  presented  as  a  group.  The  aviator  can  select 
by  touching  the  imagery  or  the  “yes”  or  “no”  softkeys. 
The  softkey  is  preferred  because  it  required  fewer  actions. 
A  single  softkey  touch  brings  up  the  next  group  of 
detections.  In  target  frames  mode  the  aviator  finds 
detection  number  19  and  processes  it.  The  aviator  then 
steps  one  frame  up  or  over.  If  there  were  a  detection,  he 
would  select  it  and  process  it.  If  there  were  more  he 
would  select  and  process  them;  otherwise,  he  would  step 
to  another  frame.  This  process  is  time  consuming  for 
several  reasons.  It  requires  a  large  amount  of  manual 
interaction  with  the  system,  it  requires  examination  of  a 
number  of  frames  containing  no  detections,  and  it 
requires  searching  the  frames  to  locate  detections. 

An  additional  factor  affecting  processing  time  in  frames 
mode  is  that  missed  target  objects  must  be  located  in  this 
mode.  This  task  is  essentially  the  same  as  the  manual 
task,  which  is  somewhat  slower  than  processing  detected 
objects.  While  this  effect  inflates  our  estimate  of  frames 
mode  processing  time  slightly,  it  is  still  the  case  that  the 
extra  work  associated  with  frames  mode  makes  it 


substantially  slower  than  chips  mode  for  processing 
detections. 

Crew-System  Interaction 

Crew-system  interaction  is  generally  considered  to  be  an 
issue  of  physical  interface  design.  Type  and  position  of 
controls,  display  formats,  operational  sequences,  and 
organization  of  data  all  affect  usability  and  effectiveness. 
We  did  not  vary  the  operator  interface,  but  implemented 
that  of  the  Comanche  ATD/C.  This  design  has  not  been 
fielded  nor  fully  tested.  We  identified  some  difficulties  in 
operating  the  aided  systems  using  this  interface.  These  are 
reported  along  with  suggested  improvements. 

A  second  crew-system  interface  issue  for  aided  targeting 
systems  is  the  criterion  for  target  detection.  For  a  given 
system  the  criterion  determines  the  ratio  of  correct 
detections  to  false  alarms.  This  ratio  must  be  optimized 
for  any  given  system  sensitivity.  The  present  study  was 
not  intended  to  optimize  criterion.  Rather  it  was  intended 
to  determine  what  aspects  of  performance  are  affected  by 
criterion  and  how  criterion  interacts  with  other  system 
design  factors. 

Since  criterion  affects  the  ratio  of  correct  detections  to 
false  alarms,  we  might  expect  it  to  affect  performance 
time.  In  fact  we  see  the  same  pattern  of  performance  time 
effects  for  criterion  that  we  saw  for  the  difference  in  aided 
system  sensitivity.  That  is,  the  lower  criterion  led  to  a 
longer  time  to  perform  the  target  hand-off  task  (fig.  3-9) 
even  though  the  mean  time  to  process  an  individual 
detection  was  shorter  (fig.  3-10).  This  pattern  occurs 
because  the  lower  criterion  generates  more  false  alarms 
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(fig.  3-11).  These  are  processed  more  rapidly  than  are 
correct  detections,  but  because  they  increase  the  total 
number  of  detections  to  be  processed,  they  lead  to  a 
longer  total  time  to  perform  the  task.  The  lower  criterion 
was  not  unequivocally  bad,  however.  Its  benefit  lay  in 
reducing  the  number  of  targets  that  were  missed  because 
they  were  never  presented  to  the  aviator  (fig.  3-12).  As 
stated  above  target  objects  were  not  presented  when  the 
system  failed  to  detect  them  and  the  aviator  failed  to 
select  the  appropriate  frame  during  his  stepwise  search.  In 
some  sense  the  lower  criterion  compensates  for  the 
cumbersome  frames  mode  interaction  by  obviating  the 
need  to  find  missed  target  objects.  The  cost  of  this 
compensation  is  a  large  increase  in  the  time  required  to 
perform  the  target  hand-off.  This  is  a  substantial  cost,  and 
the  operator  interface  should  be  optimized  to  reduce  the 
likelihood  of  failing  to  present  target  objects  prior  to 
optimizing  criterion.  Possible  alternatives  for  optimizing 
the  operator  interface  are  discussed  below. 

The  effect  of  target  density  also  turns  out  to  be  related  to 
the  aviator-system  interaction.  Increasing  the  number  of 
target  objects  in  the  environment  increases  the  target 
hand-off  time,  and  this  increase  with  target  density  is 


greatest  for  the  lower  criterion  (fig.  3-13).  The  interaction 
between  criterion  and  target  density  does  not  arise  simply 
from  there  being  more  detections  to  process,  since  neither 
the  single  sensor  nor  the  fused  sensor  system  shows  a 
sensitivity  to  target  density  relative  to  the  manual  system 
(fig.  3-14).  Rather  it  is  an  effect  of  the  operator  interface 
design.  As  mentioned  above,  detections  are  processed 
more  quickly  in  chips  mode  than  they  are  in  frames  mode. 

Thus  the  time  required  to  perform  the  hand-off  increases 
when  there  are  enough  detections  to  result  in  some  being 
presented  only  in  frames  mode.  Table  3-1  shows  the 
expected  number  of  detections  presented  only  in  frames 
mode.  All  four  target  density  conditions  result  in  frames 
mode  presentations  for  the  single  sensor,  low  criterion 
configuration.  The  highest  target  density  condition  results 
in  frames  mode  presentation  for  the  fused  sensor,  low 
criterion  configuration  and  the  single  sensor,  high 
criterion  condition.  All  of  these  conditions  show  an 
elevated  target  hand-off  time  (fig.  3-15)  except  the  no¬ 
target  object,  single  sensor  condition  and  the  single 
sensor,  high  criterion  condition. 
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Figure  3-9.  Time  to  target  hand-off  by  criterion  (P(d)  =  0.9  is  low  criterion). 
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NUMBER  OF  FALSE  ALARMS 
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Figure  3-10.  Time  to  process  individual  detections  by  criterion  (P(d)  =  0.9  is  low  criterion). 
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Figure  3-11.  Number  of  false  alarms  by  criterion  (P(d)  =  0.9  is  low  criterion). 
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Figure  3- 12.  Proportion  of  missed  target  objects  not  presented  by  criterion  (P(d)  =  0.9  is  low  criterion). 
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Figure  3- 13.  Target  hand-off  time:  interaction  between  criterion  and  target  density. 
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Figure  3- 14.  Effect  of  target  density  on  target  hand-off  time  for  each  sensor  system. 
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Table  3-1.  Number  of  detections  presented  in  frames  mode 


System 

configuration 

Number  of 
target  objects 

Number  of  non¬ 
target  objects 

Expected 
number  of 
detections 

Number 
presented  in 
frames  mode 

Fused,  P(d)  =  0.6 

0 

1 

0.6 

0 

Fused,  P(d)  =  0.6 

6 

1 

4.2 

0 

Fused,  P(d)  =  0.6 

10 

1 

6.6 

0 

Fused,  P(d)  =  0.6 

14 

1 

9.2 

0 

Fused,  P(d)  =  0.9 

0 

5 

4.5 

0 

Fused,  P(d)  =  0.9 

6 

5 

9.9 

0 

Fused,  P(d)  =  0.9 

10 

5 

13.4 

0 

Fused,  P(d)  =  0.9 

14 

5 

17.1 

0-1 

Single,  P(d)  =  0.6 

0 

6 

3.6 

0 

Single,  P(d)  =  0.6 

6 

6 

7.2 

0 

Single,  P(d)  =  0.6 

10 

6 

9.6 

0 

Single,  P(d)=  0.6 

14 

6 

12.0 

0-2 

Single,  P(d)  =  0.9 

0 

22  , 

19.8 

0-4 

Single,  P(d)  =  0.9 

6 

22 

25.2 

0-10 

Single,  P(d)  =  0.9 

10 

22 

18.8 

0-14 

Single,  P(d)  =  0.9 

14 

22 

32.4 

0-18 

Figure  3-15.  Target  hand-off  time:  interaction  between  system  configuration  and  target  density. 


The  frames  mode  effect  is  more  apparent  when  the  data 
are  segregated  by  criterion  than  when  they  are  segregated 
by  system  because  all  of  the  configurations  in  which 
detections  are  presented  in  frames  are  low  criterion 
configurations  while  only  four  out  of  five  of  these  are 
single  sensor  configurations.  The  effect  is  likely  due  to 
the  frames  mode  operator  interaction,  not  to  the  aided 
system. 

A  further  indication  of  the  ineffectiveness  of  frames 
mode  is  the  relatively  large  amount  of  time  required  to 


process  target  objects.  Even  though  the  aviators  spent 
over  twice  as  much  time  in  frames  mode  as  in  chips  mode 
(fig.  3-16),  they  processed  fewer  targets  (fig.  3-17)  in 
frames  mode.  It  is  true  that  area  clearing  and  target  list 
verification  may  inflate  the  time  spent  in  frames  mode  by 
as  much  as  70  seconds,  which  is  the  mean  time  from  the 
penultimate  target  acceptance  to  sending  the  list.  Even 
accounting  for  this  activity  as  much  time,  or  more,  is 
spent  in  frames  mode  to  detect  fewer  targets. 
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Figure  3-16.  Time  spent  in  each  system  mode. 


Figure  3- 1 7.  Percentage  of  targets  processed  in  each  system  mode. 
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Aviator  Subjective  Evaluation 

In  general  the  aviators’  subjective  evaluation  of  the 
systems  was  less  sensitive  to  differences  between  the 
systems  than  was  the  objective  data,  but  their  evaluation 
paralleled  the  objective  evaluation  (table  3-2).  They  felt 
that  the  aided  systems  provided  a  substantial  improve¬ 
ment  in  exposure  time,  although  this  improvement  led  to 
a  perceived  improvement  in  tactical  value  only  for  the 
fused  sensor  system.  They  felt  that  the  fused  system  gave 
a  shorter  target  hand-off  time  than  either  the  single  sensor 
system  or  the  manual  system  and  that  the  workload 
associated  with  the  fused  system  was  lower  than  with  the 
manual  system.  They  perceived  the  superior  detection  and 
categorization  data  provided  by  the  aided  systems,  but, 
correctly,  they  did  not  feel  that  these  resulted  in  better 
overall  performance. 


Table  3-2.  Aviator  subjective  evaluation*3 


Evaluation  factor 

Fused 

sensor 

system 

Single 

sensor 

system 

Manual 

system 

Exposure  time 

6.6 

6.6 

2.5 

System  detection 

5.4 

4.6 

2.0 

System 

categorization 

3.8 

3.2 

1.8 

Target  hand-off 
time 

5.3 

4.6 

4.1 

Workload 

5.7 

5.1 

4.8 

Tactical  value 

5.8 

5.3 

4.9 

aShows  mean  ratings.  Seven  represents  the  best  rating 
and  one  represents  the  worst. 


4.  Discussion 

Effects  of  Aiding 

The  targeting  system  evaluation  has  demonstrated  a  clear 
advantage  of  aided  systems  over  a  manual  system  having 
the  same  FLIR  sensing  capability.  The  greatest  benefit  of 
the  aided  systems  was  the  reduction  in  exposure  time. 

This  benefit  had  been  anticipated,  and  in  fact  the  exposure 
time  for  the  aided  systems  was  largely  defined  by  the 
scan  time  of  the  system.  The  relative  unknown  was  the 
exposure  time  for  the  manual  system.  At  an  average  of 
over  six  minutes,  the  exposure  time  obtained  is  longer 
than  would  be  considered  tactically  sound  under  actual 


combat  conditions,  but  it  is  an  accurate  measure  of  the 
time  needed  to  perform  the  complete  targeting  task. 
Tactical  considerations  might  force  segmentation  of  the 
task  to  reduce  exposure  time.  Such  segmentation  could 
greatly  increase  the  duration  of  a  manual  targeting  task. 
Therefore  the  benefit  derived  from  aiding  might  be  sub¬ 
stantially  greater  under  combat  conditions  than  under  the 
conditions  in  the  present  test. 

A  significant  concern  about  aided  systems  is  that  the 
improvement  in  exposure  time  might  be  purchased  at  the 
expense  of  poorer  performance.  This  was  definitely  not 
the  case.  Overall  the  aided  systems  gave  performance 
comparable  to  the  manual  system  in  accuracy  with  less 
risk  due  to  exposure  to  ground  threats.  The  fused  sensor 
system  actually  allowed  quicker  construction  of  the  target 
hand-off  list,  by  about  50  seconds,  than  either  the  single 
sensor  system  or  the  manual  system.  The  design  of  the 
test  mitigated  against  finding  accuracy  improvements 
with  the  aided  systems,  since  the  high  thermal  contrast 
(high  A  T)  image  made  the  aviator’s  visual  discrimination 
task  easy.  Aiding  could  enhance  accuracy  under  poor 
thermal  contrast  (low  A  T)  conditions.  We  might  expect 
the  greatest  benefit  from  the  fused  sensor  system,  since 
the  Longbow  would  provide  it  information  to  compensate 
for  the  poor  FLIR  data. 

Crewstation  Interface 

Chips  mode  -  The  chips  mode  interface  was  highly 
effective  in  allowing  the  aviators  to  process  detections 
quickly.  A  large  number  of  individual  detections  could  be 
processed  in  this  mode.  It  was  also  useful  for  verifying 
the  target  hand-off  list.  Its  limitation  was  that  only 
18  detections  could  be  presented  in  this  mode.  This 
limitation  was  caused  by  processing  and  memory  in 
available,  flight  qualified  hardware,  that  should  not  limit 
future  systems. 

Chips  mode  should  be  expanded  to  cover  all  system 
detections.  Chips-like  review  modes  would  also  be  useful 
for  verification  of  the  target  hand-off  list.  One  such  mode 
would  include  all  accepted  targets,  whether  detected  by 
the  system  or  by  the  crew.  This  mode  would  enable  the 
aviator  to  verify  the  accuracy  of  the  targets  he  has 
included  in  the  target  hand-off  list.  A  second  mode  would 
present  only  detections  not  accepted  by  the  crew  as 
targets.  This  mode  would  facilitate  the  aviator’s  checking 
his  own  editing. 

Chips  mode  presents  only  individual  targets  detected  by 
the  system.  A  substantial  part  of  the  reconnaissance  task 
requires  frames  mode.  Frames  mode  supports  detection  of 
target  objects  missed  by  the  system.  It  also  allows  the 
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aviator  to  view  global  relationships  between  target 
objects. 

Frames  mode-  The  aviators  were  no  more  likely  to  miss 
target  objects  with  the  aided  systems  than  with  the 
manual  system,  and  in  fact  the  miss  rate  was  very  low 
across  the  board.  There  was,  however,  a  negative  effect  in 
that  a  number  of  misses  with  the  aided  systems  resulted 
from  the  target  object  never  having  been  displayed  to  the 
aviator.  This  situation  occurred  when  the  system  failed  to 
detect  the  object  and  the  aviator  failed  to  select  the  frame 
containing  the  object  during  his  search  for  system  misses. 
This  problem  was  symptomatic  of  a  design  flaw  in  the 
operator  interface,  which  has  not  yet  been  fielded  nor 
fully  tested.  This  flaw  is  the  cumbersome  process  of 
sorting  through  a  large  number  of  sensor  frames  (92  for 
the  6°  x  45°  scan).  The  aviators  found  this  process  time 
consuming  and  monotonous.  In  addition,  the  rate  of  non- 
displayed  system  misses  indicates  that  it  is  somewhat 
error  prone. 

Further  potential  problems  exist  with  the  frames  mode 
interface  that  were  not  directly  apparent  in  the  present 
evaluation.  One  is  the  possibility  of  missing  outlying 
target  objects  due  to  failure  of  the  aviator  to  view  the 
entire  search  zone.  Appendix  C  shows  that  the  aviators 
reduced  the  area  they  actually  viewed  with  the  aided 
systems.  Another  potential  problem  is  that  relationships 
between  targets  in  the  scanned  formation  may  be  less 
apparent  when  the  area  is  viewed  in  small  frames.  The 
present  evaluation  did  not  test  this  sort  of  situation 
awareness,  but  the  aviators  commented  that  they  felt  the 
fragmented  view  to  be  less  than  satisfactory. 

Aviators  used  the  medium  field  of  view  in  the  manual 
mode  to  provide  a  big  picture  view.  Having  found 
possible  target  objects  in  the  medium  FOV,  they  switched 
to  narrow  FOV  to  gain  adequate  resolution  for  their 
decision.  Frames  mode  should  provide  a  comparable 
capability.  Pixel  expanded  medium  and  wide  FOV  modes 
would  allow  the  crew  to  search  quickly  and  thoroughly, 
getting  a  global  perspective.  Narrow  frames  would 
provide  a  detailed  image,  and  the  chips-like  modes  would 
support  review  of  the  final  product  of  the  reconnaissance 
activity. 


act  on  the  information  provided  to  him  by  the  system. 

As  criterion  varies,  the  aviator  has  to  deal  with  varying 
numbers  of  hits  and  false  alarms.  We  have  seen  that  the 
system’s  judgment  does  little  to  bias  the  aviator,  at  least 
under  the  high  thermal  contrast  conditions  of  the  present 
evaluation.  The  principal  effect  of  criterion  variation  is  to 
change  the  amount  of  frames  mode  interaction.  Frames 
mode  can  lead  to  failure  to  present  target  objects  when 
criterion  is  high.  When  criterion  is  low,  frames  mode 
increases  the  time  needed  to  perform  the  reconnaissance. 
Both  of  these  effects  stem  from  the  design  problems  cited 
above.  A  definitive  judgment  regarding  criterion  values 
must  await  optimization  of  system  modes. 

Another  interaction  factor  linked  to  the  criterion  issue  is 
the  crew’s  response  to  detected  non-target  objects.  We 
required  an  explicit  response  for  data  collection.  This 
response  slowed  processing  of  detected  non-target 
objects.  A  “no-target”  default  would  speed  the  task  and 
would  likely  have  no  adverse  effect  on  accuracy.  This 
change  should  be  included  in  an  operator  interface 
refinement  and  evaluation  program. 

5.  Conclusions 

1.  Aided  targeting  systems  provide  a  substantial  reduction 
(over  8 1  %)  in  exposure  to  ground  based  threats. 

2.  The  fused  sensor  system  provides  a  considerable 
reduction  in  time  to  construct  the  target  hand-off  list. 

3.  The  benefits  of  aiding  are  obtained  with  no  loss  of 
accuracy.  The  fused  sensor  system  actually  showed 
slightly  lower  false  alarm  rates  than  the  other  systems. 

4.  Aiding  may  improve  accuracy  under  poor  FLIR 
viewing  conditions  (low  A  T),  particularly  with  the  fused 
sensor  system. 

5.  Battlefield  conditions,  which  lead  to  greater  time  stress, 
may  magnify  the  benefits  obtained  under  the  present, 
relatively  benign  test  conditions. 

6.  Operator  interface/system  mode  refinement  and 
evaluation  should  be  an  integral  part  of  system  design  to 
address  problem  areas  identified  in  the  present  test. 


Crew-System  Interaction 

We  have  addressed  crew-system  interaction  as  a  criterion 
issue.  Detection  criterion  affects  how  the  aviator  has  to 
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6.  Appendices 


A.  Aviator  Subjective  Evaluation  Form 


Targeting  System  Evaluation  Questionnaire 


Name: 

Run  number: 

1 .  Rate  the  ease  of  use  of  each  targeting  svstem. 

Manual  1  2 

3 

4 

5 

6 

7 

Single- 

Sensor 

Multi- 

very  hard 

1 

2 

.3 

4 

5 

6 

very  easy 

7 

very  hard 

1 

2 

3 

4 

5 

6 

very  easy 

7 

Sensor 

2.  Rate  the  accuracy 

very  hard 

of  each  targeting  svstem. 

very  easy 

Manual 

1 

2 

.3 

4 

5 

6 

7 

Single- 

Sensor 

Multi- 

very  poor 

1 

2 

.3 

4 

5 

6 

very  good 

7 

very  poor 

1 

2 

3 

4 

5 

6 

very  good 

7 

Sensor  very  poor 

3.  Rate  vour  accuracy  using  each  targeting  svstem. 

very  good 

Manual 

1 

2 

3 

4 

5 

6 

7 

Single- 

Sensor 

Multi- 

very  poor 

1 

2 

3 

4 

5 

6 

very  good 

7 

very  poor 

1 

2 

3 

4 

5 

6 

very  good 

7 

Sensor  very  poor 

4  Rate  exposure  time  with  each  targeting  svstem. 

very  good 

Manual 

1 

2 

3 

4 

5 

6 

7 

Single- 

Sensor 

Multi- 

very  long 

1 

2 

3 

4 

5 

6 

very  long 

1 

2 

3 

4 

5 

6 

very  short 

7 

Sensor 

5.  Rate  total  time  to 

very  long 

nerform  targeting  task. 

very  short 

Manual 

1 

2 

3 

4 

5 

6 

7 

Single- 

Sensor 

Multi- 

very  long 

1 

2 

3 

4 

5 

6 

very  short 

7 

very  long 

1 

2 

3 

4 

5 

6 

very  short 

7 

Sensor  very  long 

6.  Rate  the  tactical  value  of  each  svstem. 

very  short 

Manual 

1 

2 

3 

4 

5 

6 

7 

Single- 

Sensor 

Multi- 

very  low 

1 

2 

3 

4 

5 

6 

very  high 

7 

very  low 

1 

2 

3 

4 

5 

6 

very  high 

7 

Sensor 

very  low 

very  high 

B.  Analysis  of  Variance  Tables 


Number  of  observations  in  data  set  =  422 
Dependent  Variable:  EXPSD  Total  Time  Aircraft  was  Exposed  (seconds) 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Value 

Pr  >  F 

Model 

14 

3227380.10541434 

230527.15038674 

88.87 

0.0001 

Error 

407 

1055726.26571824 

2593.92202879 

Corrected  Total 

421 

4283106.37113258 

R-Square 

C.V. 

Root  MSE 

EXPSD  Mean 

0.753514 

56.66843 

50.93056085 

89.87465599 

Source 

DF 

Type  IE  SS 

Mean  Square 

F  Value 

Pr  >  F 

PILOT 

10 

83536.94273476 

8353.69427348 

3.22 

0.0005 

TGT_M 

4 

3133994.89815930 

783498.72453983 

302.05 

0.0001 

Bl.  ANOVA — Exposure  Time. 


Number  of  observations  in  data  set  =  422 
Dependent  Variable:  N_R_N_D  Number  Real  Targets  NOT  Detected 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Value 

Pr  >  F 

Model 

121 

13.66991075 

0.11297447 

0.66 

0.9936 

Error 

190 

32.67624310 

0.17198023 

Corrected  Total 

311 

46.34615385 

R-Square 

C.V. 

Root  MSE 

N_R_N_D  Mean 

0.294952 

308.0666 

0.41470499 

0.13461538 

Source 

DF 

Type  IE  SS 

Mean  Square 

F  Value 

Pr>F 

PILOT 

10 

1.28089976 

0.12808998 

0.74 

0.6816 

RUN 

39 

4.01890639 

0.10304888 

0.60 

0.9703 

TGT_M_J3 

2 

0.11084093 

0.05542047 

0.32 

0.7249 

RUN*TGT_M_J3 

70 

7.50981624 

0.10728309 

0.62 

0.9881 

B2.  ANOVA — Misses  by  Targeting  System. 


Number  of  observations  in  data  set  =  422 
NOTE:  Due  to  missing  values,  only  421  observations  can  be  used  in  this  analysis. 
Dependent  Variable:  N_F_ACC  Number  False  Targets  Accepted  as  Real 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Value  Pr  >  F 

Model 

126 

692.54453316 

5.49638518 

4.16  0.0001 

Error 

294 

388.37233145 

1 .32099432 

Corrected  Total 

420 

1080.91686461 

R-Square 

C.V. 

Root  MSE 

N_F_ACC  Mean 

0.640701 

140.6611 

1.14934517 

0.81710214 

DF  Type  III  SS  Mean  Square  F  Value  Pr  >  F 

10  220.81298601  22.08129860  16.72  0.0001 

39  105.03549591  2.69321784  2.04  0.0005 

2  90.94249257  45.47124628  34.42  0.0001 

75  147.18956166  1.96252749  1.49  0.0113 

B3.  ANOVA — False  Alarms. 


Number  of  observations  in  data  set  =  422 
Dependent  Variable:  PROP_OF_MIS SED_T ARGETS_N OT_PRESENTED 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Value  Pr  >  F 

Model 

12 

0.33071382 

0.02755949 

0.85  0.5949 

Error 

409 

13.20483120 

0.03228565 

Corrected  Total 

421 

13.53554502 

R-Square 

C.V. 

Root  MSE 

PROP_M  Mean 

0.024433 

541.6131 

0.17968208 

0.03:11753 

Source 

DF 

Type  IE  SS 

Mean  Square 

F  Value  Pr  >  F 

PILOT 

10 

0.09808461 

0.00980846 

0.30  0.9801 

TGT_M_J3 

2 

0.22915471 

0.11457735 

3.55  0.0296 

B4.  ANOVA — Number  of  Missed  Target  Objects  not  Presented. 


Source 

PILOT 

RUN 

TGT_M_J3 

RUN*TGT_M_J3 
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Number  of  observations  in  data  set  =  422 

Dependent  Variable:  TTH  Time  To  Target  Hand-off  (seconds) 

Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Value 

Pr>F 

Model 

12 

1242489.51191388 

103540.79265949 

11.16 

0.0001 

Error 

409 

3793218.76751411 

9274.37351470 

Corrected  Total 

421 

5035708.27942799 

R-Square 

C.V. 

Root  MSE 

TTH  Mean 

0.246736 

39.92129 

96.30354882 

241.23358899 

Source 

DF 

Type  III  SS 

Mean  Square 

F  Value 

Pr  >  F 

PILOT 

10 

864300.21394836 

86430.02139484 

9.32 

0.0001 

TGT_M_J3 

2 

363255.23426774 

181627.61713387 

19.58 

0.0001 

B5.  AN OVA — Time  to  Target  Hand-off. 


Number  of  observations  in  data  set  =  422 
Dependent  Variable:  TIME  Time  Required  For  Target  Determination 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Value 

Pr  >  F 

Model 

12 

24760.95770457 

2063.41314205 

51.17 

0.0001 

Error 

5614 

226377.83178860 

40.32380331 

Corrected  Total 

5626 

251138.78949317 

R-Square 

C.V. 

Root  MSE 

TIME  Mean 

0.098595 

80.28132 

6.35010262 

7.90981300 

Source 

DF 

Type  in  SS 

Mean  Square 

F  Value 

Pr  >  F 

PILOT 

10 

17898.76238181 

1789.87623818 

44.39 

0.0001 

TGT_M_J3 

2 

6187.06434307 

3093.53217154 

76.72 

0.0001 

B6.  ANOVA—Time  to  Process  an  Individual  Target. 
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Number  of  observations  in  data  set  =  422 
Dependent  Variable:  TIME  Time  Required  For  Target  Determination 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Value 

Pr  >  F 

Model 

15 

65090.45764067 

4339.36384271 

130.87 

0.0001 

Error 

5611 

186048.33185249 

33.15778504 

Corrected  Total 

5626 

251138.78949317 

R-Square 

C.V. 

Root  MSE 

TIME  Mean 

0.259181 

72.79919 

5.75827969 

7.90981300 

Source 

DF 

Type  III  SS 

Mean  Square 

F  Value 

Pr  >  F 

PILOT 

10 

13804.11564181 

1380.41156418 

41.63 

0.0001 

REAL 

1 

8193.69244699 

8193.69244699 

247.11 

0.0001 

MODE 

2 

9630.56492595 

4815.28246297 

145.22 

0.0001 

MODE*REAL 

2 

3517.53880691 

1758.76940345 

53.04 

0.0001 

B7.  ANOVA — Time  to  Enter  Individual  Target  Data. 


Number  of  observations  in  data  set  =  422 
Dependent  Variable:  TTH  Time  To  Target  Hand-off  (seconds) 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Value 

Pr  >  F 

Model 

13 

1399191.52145124 

107630.11703471 

13.03 

0.0001 

Error 

321 

2650511.78737057 

8257.04606657 

Corrected  Total 

334 

4049703.30882181 

R-Square 

C.V. 

Root  MSE 

TTH  Mean 

0.345505 

38.38302 

90.86828966 

236.74083826 

Source 

DF 

Type IH  SS 

Mean  Square 

F  Value 

Pr  >  F 

PILOT 

10 

618816.03621018 

61881.60362102 

7.49 

0.0001 

ATM_C 

1 

324094.27126135 

324094.27126135 

39.25 

0.0001 

ATM_S 

1 

333486.26219446 

333486.26219446 

4039 

0.0001 

ATM_C*ATM_S 

1 

89049.35866618 

89049.35866618 

10.78 

0.0011 

B8.  ANOVA — Time  to  Target  Hand-off  by  Criterion. 
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Number  of  observations  in  data  set  =  422 
Dependent  Variable:  TIME  Time  Required  For  Target  Determination 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Value 

Pr  >  F 

Model 

14 

53336.69941862 

3809.76424419 

115.34 

0.0001 

Error 

4819 

159179.00186784 

33.03154220 

Corrected  Total 

4833 

212515.70128646 

R-Square 

C.V. 

Root  MSE 

TIME  Mean 

0.250978 

76.17454 

5.74730739 

7.54491857 

Source 

DF 

Type  III  SS 

Mean  Square 

F  Value 

Pr  >  F 

PILOT 

10 

18655.61402806 

1865.56140281 

56.48 

0.0001 

TM_C 

1 

18.40346889 

18.40346889 

0.56 

0.4554 

ATM_S 

1 

48.27414799 

48.27414799 

1.46 

0.2268 

ATM_C*ATM_S 

1 

103.59856558 

103.59856558 

3.14 

0.0766 

REAL 

1 

29092.16932434 

29092.16932434 

880.74 

0.0001 

B9.  ANOVA—Time  to  Process  Detections  by  System  and  Criterion. 


Number  of  observations  in  data  set  =  422 
Dependent  Variable:  N_F_ACC  Number  False  Targets  Accepted  as  Real 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Value 

Pr  >  F 

Model 

52 

375.81741717 

7.22725802 

6.51 

0.0001 

Error 

281 

311.83827146 

1.10974474 

Corrected  Total 

333 

687.65568862 

R-Square 

C.V. 

Root  MSE 

N_F_ACC  Mean 

0.546520 

149.7236 

1.05344423 

0.70359281 

Source 

DF 

Type  lH  SS 

Mean  Square 

F  Value 

Pr  >  F 

PILOT 

10 

163.22601576 

16.32260158 

14.71 

0.0001 

RUN 

39 

70.73831310 

1.81380290 

1.63 

0.0133 

ATM_C 

1 

30.80365069 

30.80365069 

27.76 

0.0001 

ATM_S 

1 

79.53399499 

79.53399499 

71.67 

0.0001 

ATM_C*ATM_S 

1 

8.91574605 

8.91574605 

8.03 

0.0049 

BIO.  ANOVA— False  Alarms  by  System  and  Criterion. 


Number  of  observations  in  data  set  =  422 
Dependent  Variable:  TTH  Time  To  Target  Hand-off  (seconds) 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Value 

Pr  >  F 

Model 

21 

2776553.81415053 

132216.84829288 

23.41 

0.0001 

Error 

400 

2259154.46527746 

5647.88616319 

Corrected  Total 

421 

5035708.27942799 

R-Square 

C.V. 

Root  MSE 

TTH  Mean 

0.551373 

31.15338 

75.15241954 

241.23358899 

Source 

DF 

Type  III  SS 

Mean  Square 

F  Value 

Pr  >  F 

PILOT 

10 

897691.12591794 

89769.11259179 

15.89 

0.0001 

TGT_DENS 

3 

1306729.71218274 

435576.57072758 

77.12 

0.0001 

TGT_M_J3 

2 

339009.91596192 

169504.95798096 

30.01 

0.0001 

TGT_M_J3 

6 

52553.68400731 

8758.94733455 

1.55 

0.1603 

*TGT_DENS 

Bll.  ANOVA — Effect  of  Target  Density. 


Number  of  observations  in  data  set  =  422 
Dependent  Variable:  TTH  Time  To  Target  Hand-off  (seconds) 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Value 

Pr  >  F 

Model 

25 

2726648.50579628 

109065.94023185 

25.47 

0.0001 

Error 

309 

1323054.80302553 

4281.73075413 

Corrected  Total 

334 

4049703.30882181 

R-Square 

C.V. 

Root  MSE 

TTH  Mean 

0.673296 

27.63990 

65.43493527 

236.74083826 

Source 

DF 

Type  in  SS 

Mean  Square 

F  Value 

Pr  >  F 

PILOT 

10 

646497.28869393 

64649.72886939 

15.10 

0.0001 

TGT_DENS 

3 

1248183.65886381 

416061.21962127 

97.17 

0.0001 

TGT_M 

3 

741285.86665516 

247095.28888505 

57.71 

0.0001 

TGT_M*TGT_DENS 

9 

72619.05200043 

8068.78355560 

1.88 

0.0537 

B1 2.  ANOVA — Effect  of  Target  Density  by  Aided  System  Configuration. 


Number  of  observations  in  data  set  =  422 
Dependent  Variable:  TIMEM  Time  Spent  In  Mode  (seconds) 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Value 

Pr  >  F 

Model 

34 

10170726.80218550 

299139.02359369 

169.65 

0.0001 

Error 

2060 

3632420.40672497 

763.31087705 

Corrected  Total 

2094 

13803147.20891050 

R-Square 

C.V. 

Root  MSE 

TIMEM  Mean 

0.736841 

96.04019 

41.99179535 

43.72314849 

Source 

DF 

Type  III  SS 

Mean  Square 

F  Value 

Pr  >  F 

PILOT 

10 

153765.38166671 

15376.53816667 

8.72 

0.0001 

MODE 

4 

3489486.42048208 

872371.60512052 

494.73 

0.0001 

TGT_M 

4 

158064.56859834 

39516.14214958 

22.41 

0.0001 

MODE*TGT_M 

16 

6377055.47521187 

398565.96720074 

226.03 

0.0001 

B13.  AN OVA — Time  Spent  in  Mode. 

Source 

SS 

df 

MS  F 

P 

Between  Blocks 

9.0626 

11 

Within  Blocks 

147.8172 

24 

SENSOR 

130.0482 

2 

65.0241  80.507 

0.0000 

Error 

17.7690 

22 

0.8077 

Total 

156.8798 

35 

B14.  AN OVA— Subjective  Evaluation  of  Exposure  Time. 

Source 

SS 

df 

MS 

F 

P 

Between  Blocks 

24.0024 

11 

Within  Blocks 

117.2898 

24 

SENSOR 

77.4934 

2 

38.7467 

21.420 

0.0000 

Error 

17.7690 

22 

0.8077 

Total 

141.2922 

35 

B15.  AN  OVA— Subjective  Evaluation  of  System  Detection. 
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Source 

SS 

df 

MS 

F 

P 

Between  Blocks 

18.4040 

1 1 

Within  Blocks 

57.4687 

24 

SENSOR 

23.6204 

2 

11.8102 

7.676 

0.0033 

Error 

33.8483 

22 

1.5386 

Total 

75.8727 

35 

B16.  ANOVA- 

—Subjective  Evaluation  of  System  Categorization. 

Source 

SS 

df 

MS 

F 

P 

Between  Blocks 

20.4431 

11 

Within  Blocks 

14.2856 

24 

SENSOR 

4.2539 

2 

2.1264 

4.663 

0.0202 

Error 

10.0327 

22 

0.4560 

Total 

34.7286 

35 

B17.  ANOVA- 

—Subjective  Evaluation  of  Workload. 

Source 

SS 

df 

MS 

F 

P 

Between  Blocks 

18.3413 

11 

Within  Blocks 

15.8502 

24 

SENSOR 

5.0052 

2 

2.5026 

5.077 

0.0152 

Error 

10.8450 

22 

0.4930 

Total 

34.1915 

35 

B18.  ANOVA — Subjective  Evaluation  of  Tactical  Value. 
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C.  Sample  Sensor  Scans 


Vista#:  3  Heading  :  331.0 


C.6.  Aided  Sensor  Scan.  Grid  Units — 2°inAZ,  1.5°  in  EL. 
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interface  for  both  was  like  that  of  an  advanced  scout  helicopter  aided  mode.  Exposure  time  while  performing 
the  task  was  reduced  substantially  with  the  aided  systems,  with  no  loss  of  target  hand-off  list  accuracy.  The 
fused  sensor  system  showed  lower  time  to  construct  the  target  hand-off  list  and  a  slightly  lower  false  alarm 
rate  then  the  other  systems.  A  number  of  issues  regarding  system  sensitivity  and  criterion,  and  operator 
interface  design  are  discussed. 
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